Apparatus and System for Recognizing Environment Surrounding Vehicle

ABSTRACT

In conventional systems using an onboard camera disposed rearward of a vehicle for recognizing an object surrounding the vehicle, the object is recognized by the camera disposed rearward of the vehicle. In the image recognized by the camera, a road surface marking taken by the camera appears at a lower end of a screen of the image, which makes it difficult to predict a specific position in the screen from which the road surface marking appears. Further, an angle of depression of the camera is large, and it is a short period of time to acquire the object. Therefore, it is difficult to improve a recognition rate and to reduce false recognition. Results of recognition (type, position, angle, recognition time) made by a camera disposed forward of the vehicle, are used to predict a specific timing and a specific position of a field of view of a camera disposed rearward of the vehicle, at which the object appears. Parameters of recognition logic of the rearwardly disposed camera and processing timing are then optimally adjusted. Further, luminance information of the image from the forwardly disposed camera is used to predict possible changes to be made in luminance of the field of view of the rearwardly disposed camera. Gain and exposure time of the rearwardly disposed camera are then adjusted.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The specification relates to an apparatus and system that process imagestaken by a camera mounted in a vehicle and recognize road surfacemarkings, traffic lights, signs, and the like on roads that surround thevehicle.

2. Description of the Related Art

JP-A-3-220410 discloses an apparatus that processes images taken by acamera mounted in a vehicle and recognizes road surface markings.

JP-A-6-206196 discloses an apparatus including cameras disposedforwardly and rearwardly of a vehicle. The apparatus detects contrast ofan image taken by the forward camera and, if it is hard to obtaininformation on a forward image, uses information obtained by therearward camera to recognize environment surrounding the vehicle.

SUMMARY OF THE INVENTION

The apparatus disclosed in JP-A-3-220410 has the onboard camera disposedrearwardly of the vehicle for recognizing the road surface markings. Theroad surface marking appears from a lower end of a screen of an imagetaken by the camera, which makes it difficult to predict a specificposition in the screen from which the road surface marking appears. Inaddition, the camera has a large angle of depression at an installationposition thereof, so that only a narrow portion of the road surfacefalls within a field of view of the camera. It is therefore an extremelyshort period of time for an object to be recognized to be acquired.Accordingly, it is difficult to improve a recognition rate and reducefalse recognition.

The apparatus disclosed in JP-A-6-206196, on the other hand, selectseither the forward camera or the rearward camera, active at one time,and there is no data exchange taking place between the forward andrearward cameras. The two cameras are not thus utilized effectively.There is therefore room for further improving the recognition rate andreduce false recognition.

In a system recognizing an object to be recognized, such as a roadsurface marking or the like, by processing an image taken by a firstcamera disposed rearwardly of a vehicle, results of recognition (type,position, and angle of the object of interest recognized, and a time ofrecognition) made by a second camera, such as a camera or the likedisposed forwardly of the vehicle, are used to predict a specific timingand a specific position of a field of view of the rearwardly disposedfirst camera, at which the object to be recognized appears. Parameters(a recognition area, a threshold value for extracting a characteristicquantity, and the like) of recognition logic of the rearwardly disposedfirst camera and processing timing are then adjusted.

Luminance information of the image taken by the second camera, such as acamera or the like disposed forwardly of the vehicle, is used to predictpossible changes to be made in luminance of the field of view of therearwardly disposed first camera. Gain and exposure time of therearwardly disposed first camera are then adjusted. Parameters (gain andexposure time) of the first camera are thereby adjusted even morequickly, so that even more accurate recognition of the object to berecognized can be achieved.

An improved recognition rate of the object to be recognized and reducedfalse recognition can be achieved as compared with the apparatus usingonly a single camera.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view illustrating an embodiment.

FIG. 2 is a view illustrating the embodiment.

FIG. 3 is a flowchart showing processes executed by a road surfacemarking recognition function of a rear camera image recognitionfunction.

FIG. 4 is a view illustrating an image pre-process in the flowchartshown in FIG. 2.

FIG. 5 is a view illustrating a road surface marking characteristicquantity extraction process in the flowchart shown in FIG. 2.

FIG. 6 is a flowchart showing processes executed by a recognition methodevaluation function of the rear camera image recognition function.

FIG. 7 is a detailed flowchart showing a rear camera gain valuedetermination process of the recognition method evaluation function ofthe rear camera image recognition function.

FIG. 8 shows a rear camera gain value schedule table.

FIG. 9 is a flowchart showing processes executed by a rear cameracontrol section of the rear camera image recognition function.

FIG. 10 is a view illustrating a method of representing a position of ashadow in a front camera.

FIG. 11 is a flowchart showing the processes performed by a front imagerecognition function.

FIG. 12 is a flowchart showing processes performed in an image luminancestatistical process of the front camera image recognition function.

FIG. 13 is a view showing image luminance accumulated data.

FIG. 14 is a view illustrating a method for acquiring image luminance.

FIG. 15 shows an image luminance statistical table.

FIG. 16 is a view illustrating a method for determining a road surfacecondition.

FIG. 17 is a view illustrating a method for determining a road surfacecondition.

FIG. 18 is a view illustrating a method for acquiring image luminance.

FIG. 19 is a flowchart showing processes performed in a shadow positionrecognition process of the front camera image recognition function.

FIG. 20 is a view illustrating an image coordinate system of the frontcamera.

FIG. 21 shows a conversion table for conversion between a roadcoordinate system and a screen coordinate system.

FIG. 22 is a view illustrating the road coordinate system.

FIG. 23 shows front camera shadow position data.

FIG. 24 is a flowchart showing a rear camera shadow appearanceestimation process of a rear camera exposure time determination processas part of the recognition method evaluation function of the rear cameraimage recognition function.

FIGS. 25A and 25B are views illustrating a road surface markingrecognition area of the rear camera.

FIG. 26 shows rear camera shadow position data.

FIG. 27 shows a table of image luminance and gain values.

FIG. 28 is a flowchart showing the rear camera exposure timedetermination process as part of the recognition method evaluationfunction of the rear camera image recognition function.

FIG. 29 shows a rear camera exposure time schedule tables.

FIG. 30 is a flowchart showing processes performed by the rear cameracontrol section of the rear camera image recognition function.

FIG. 31 shows a table of image luminance and exposure time.

FIG. 32 is a flowchart showing a rear camera object recognitiondetermination process as part of the recognition method evaluationfunction of the rear camera image recognition function.

FIG. 33 is a view illustrating the position of an object recognized bythe front camera and the angle of the object recognized relative to thevehicle.

FIG. 34 is a view illustrating the position of a white line recognizedby the front camera and the angle of the white line relative to thevehicle.

FIG. 35 shows nearby road surface marking data.

FIG. 36 shows front camera recognition result data.

FIG. 37 shows data on an object to be recognized by the rear camera.

FIG. 38 is a flowchart showing a rear camera process timingdetermination process as part of the recognition method evaluationfunction of the rear camera image recognition function.

FIG. 39 is a view defining the position of an object to be recognized bythe rear camera and the angle of the same relative to the vehicle.

FIG. 40 is a view illustrating the position of a white line and theangle thereof relative to the vehicle.

FIG. 41 is a view showing a positional relationship among a field ofview of the front camera, a field of view of the rear camera, and thevehicle.

FIG. 42 is a flowchart showing a rear camera recognition logic parameterdetermination process as part of the recognition method evaluationfunction of the rear camera image recognition function.

FIG. 43 is a view illustrating adjustment of the rear camera recognitionarea.

FIG. 44 shows a conversion table for conversion between the roadcoordinate system and the rear camera screen coordinate system.

FIG. 45 is a view illustrating a method for determining a characteristicquantity extraction threshold value.

FIG. 46 is a view illustrating a method for determining thecharacteristic quantity extraction threshold value.

FIG. 47 shows recognition parameter data.

FIG. 48 shows a block diagram of a system for recognizing an environmentsurrounding a vehicle according to a second embodiment.

FIG. 49 shows a block diagram of a system for recognizing an environmentsurrounding a vehicle according to a third embodiment.

FIG. 50 shows a block diagram of a system for recognizing an environmentsurrounding a vehicle according to a fourth embodiment.

FIG. 51 shows a block diagram of a system for recognizing an environmentsurrounding a vehicle according to a fifth embodiment.

FIG. 52 shows a block diagram of a system for recognizing an environmentsurrounding a vehicle according to a sixth embodiment.

FIG. 53 shows a block diagram of a system for recognizing an environmentsurrounding a vehicle according to a seventh embodiment.

FIG. 54 shows a block diagram of a system for recognizing an environmentsurrounding a vehicle according to the seventh embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments will be described below with reference to theaccompanying drawings.

First Embodiment

A first embodiment will be described as applied to a system recognizingroad surface markings by using images taken by a rear camera mounted ina vehicle.

FIG. 1 is a view showing a system for recognizing an environmentsurrounding a vehicle according to the first embodiment. A vehicle 1 hasa front camera 101 and a rear camera 108 mounted thereon. The frontcamera 101 takes an image of a view forward of the vehicle 1. The rearcamera 108 takes an image of a view rearward of the vehicle 1. The frontcamera 101 is mounted such that a road surface falls within a field ofview 4 of the front camera 101. The rear camera 108 is mounted such thatthe road surface falls within a field of view 5 of the rear camera 108.

Image information captured by the front camera 101 is inputted to asurrounding environment recognition apparatus 2. The surroundingenvironment recognition apparatus 2 recognizes a road surface marking 3a forward of the vehicle 1 based on the image information inputtedthereto. It is to be noted that an arrangement may be made to recognizewhite lines 3 d, 3 e, a sign 3 c, and a traffic light (not shown). Roadsurface markings, white lines, signs, and traffic signals willhereinafter be collectively referred to as “road surface marking or thelike.” The “road surface marking” refers to traffic signs marked on theroad, typically including pedestrian crossings, stop lines, maximumspeed limit markings, follow directions, and no U-turn markings.

Similarly, image information captured by the rear camera 108 is inputtedto the surrounding environment recognition apparatus 2. The surroundingenvironment recognition apparatus 2 recognizes a road surface marking 3b rearward of the vehicle 1 based on the image information inputtedthereto. The surrounding environment recognition apparatus 2 transmitsinformation on the road surface markings to a vehicle control apparatus106 a or an onboard information apparatus 106 b via a communicationpath. It is to be noted herein that the vehicle control apparatus 106 amay typically be a cruise control apparatus, a headway controlapparatus, or a traffic congestion follow-up control apparatus forcontrolling acceleration and deceleration of the vehicle according tothe surrounding environment. The vehicle control apparatus 106 aprovides control in accordance with the information on the road surfacemarkings transmitted from the surrounding environment recognitionapparatus 2. The onboard information apparatus 106 b, on the other hand,may typically be a navigation apparatus that corrects the position of ahost vehicle based on the information on the road surface markingstransmitted from the surrounding environment recognition apparatus 2.

FIG. 2 is a functional block diagram of the aforementioned surroundingenvironment recognition apparatus 2. The surrounding environmentrecognition apparatus 2 includes a front camera image recognition unit102 and a rear camera image recognition unit 103.

The front camera image recognition unit 102 includes a front roadsurface marking recognition section 102 a and a front camera controlsection 102 b. The front road surface marking recognition section 102 arecognizes the road surface marking or the like from the imageinformation captured by the front camera 101. The front camera controlsection 102 b controls imaging conditions (timing, cycle, exposure time,zoom, and the like) of the front camera 101. The front camera controlsection 102 b may be omitted if the imaging conditions of the frontcamera 101 are fixed.

The front road surface marking recognition section 102 a recognizes theroad surface marking or the like by performing image processing,including binarization, edge extraction, pattern matching, and the like,for the image information captured by the front camera 101.Specifically, the front road surface marking recognition section 102 adetects the type, position, angle, and the like of the road surfacemarking or the like in the image.

The rear camera image recognition unit 103 includes a rear road surfacemarking recognition section 105, a rear camera control section 107, anda recognition method evaluation section 104. The rear road surfacemarking recognition section 105 recognizes the road surface marking orthe like from the image information captured by the rear camera 108. Therear camera control section 107 controls imaging conditions (timing,cycle, exposure time, zoom, and the like) of the rear camera 108. Therecognition method evaluation section 104, on the other hand, determinesthe imaging conditions of the rear camera 108 and specific details ofimage processing performed in the rear road surface marking recognitionsection 105 based on the information inputted from the front roadsurface marking recognition section 102 a. The recognition methodevaluation section 104 then transmits information to the rear cameracontrol section 107 and the rear road surface marking recognitionsection 105.

Instead of directly inputting results of recognition made by the frontroad surface marking recognition section 102 a to the recognition methodevaluation section 104 as described above, it may still be arranged sothat the results of recognition are stored in a road surface markinginformation storage section 111 and the stored results are inputted tothe recognition method evaluation section 104. Such arrangements allowcommunication timing and processing timing to be adjusted. It is furtherpossible to identify differences among a plurality of images taken atdifferent timings, so that information on changes in the surroundingenvironment with time can be transmitted to the recognition methodevaluation section 104.

The rear camera control section 107 captures an image by controlling therear camera 108 using conditions specified by the recognition methodevaluation section 104. The rear road surface marking recognitionsection 105 recognizes the road surface marking or the like byperforming the image processing specified by the recognition methodevaluation section 104 for the image information captured under theforegoing conditions. Specifically, the rear road surface markingrecognition section 105 detects the type, position, angle, and the likeof the road surface marking or the like in the image.

Processing performed by the aforementioned rear camera image recognitionunit 103 will be described below with reference to FIGS. 3 and 6.

FIG. 3 is a flowchart showing processes executed by the rear roadsurface marking recognition section 105.

An image taken by the rear camera 108 is acquired in step 201 ofperforming an image input process.

In step 207 of selecting an object to be recognized, data 1906 on anobject to be recognized by the rear camera is read and the type of theroad surface marking or the like to be recognized by the rear camera 108is extracted.

The data 1906 on the object to be recognized by the rear camera islisted in a table of FIG. 37. Referring to FIG. 37, reference numeral2202 represents a type of object to be recognized. Reference numeral2203 represents an estimated time of appearance at which the object tobe recognized appears in the field of view of the rear camera 108. Thedata 1906 on the object to be recognized by the rear camera is createdby the recognition method evaluation section 104 based on front camerarecognition result data 806 (FIG. 36) inputted from the front roadsurface marking recognition section 102 a. A method of creating the data1906 on the object to be recognized by the rear camera will be describedlater. In step 207, a current time is referred to in the table of FIG.37 and, if it is found that the current time reaches the estimated timeof appearance 2203, the object to be recognized in question is extractedas the object to be recognized.

Processes for recognizing each of the road surface markings to berecognized will next be performed in steps 202 through 205.

Specifically, in step 202 of performing an image pre-process, noise isremoved from the image acquired in the step 201 of performing the imageinput process. The noise removal process is not mandatory for thepresent invention. It is, however, desirable that the noise removalprocess be performed since noise can very often be a hindrance torecognition of the road surface marking or the like. Noise of variouskinds can be conceivable. The objects of interest to be recognized inthe first embodiment are the road surface markings and white lines onthe road. Understandably, therefore, a problem that needs specialattention is noise arising from a “thin spot” in paint. In accordancewith the first embodiment, a process is performed for removing the “thinspot” of the road surface marking paint, so that characteristicquantities of the road surface markings and white lines can be moreeasily extracted. Typically, the thin spot removal process includes thefollowing method. Specifically, referring to FIG. 4, of luminance valuesof pixels of 3-by-3 regions f0 to f8 adjoining around a specific pixelf4 (212) of an input screen 208, the maximum luminance value is replacedas the luminance value of the specific pixel f4. This results in theluminance value of the brightest portion of the surrounding pixelsbecoming the brightness of the specific pixel, so that the thin spot canbe corrected.

In step 203 of performing a road surface marking characteristic quantityextraction process, a change in the luminance value between a roadsurface 209 and a road surface marking 210 in the input screen 208 ofFIG. 5 is detected to extract an outline of the road surface marking210. At this time, the outline of the road surface marking 210 to berecognized is detected by reading recognition parameter data 2406, andreferring to recognition areas 2433, 2434, an estimated value 2435 ofthe change in the luminance value between the road surface and the roadsurface marking, and an estimated value 2436 of the maximum value of theroad surface marking luminance value shown in FIG. 47. The recognitionparameter data 2406 defines a position on the image of the rear camera108, at which the road surface marking or the like to be recognized isexpected to appear, and an threshold value and the like of edgeextraction for recognizing the road surface marking or the like. Therecognition parameter data 2406 is created by the recognition methodevaluation section 104 based on the front camera recognition result data806 (FIG. 36) inputted from the front road surface marking recognitionsection 102 a. A specific method for creating the recognition parameterdata 2406 will be described in detail later.

In step 204 of performing a determination process, it is determinedwhether or not the outline of the road surface marking 210 extracted instep 203 of performing the road surface marking characteristic quantityextraction process coincides with characteristics of the road surfacemarking of interest selected in step 207 of performing the object ofinterest to be recognized selection process. Specifically, the rear roadsurface marking recognition section 105 has template data correspondingto outlines of the road surface markings or the like that are expectedto occur. The outline of the road surface marking or the like recognizedfrom the image taken by the rear camera 108 is compared with thetemplate data. If there is found a match between the outline and thetemplate data, or if a difference between the two falls within apredetermined range, it is determined that the road surface marking inquestion is recognized. Alternatively, an arrangement may be made, inwhich the recognition method evaluation section 104 attaches templatedata required when sending the front camera recognition result data 806.

In step 205 of performing a recognition result output process, if theroad surface marking to be recognized has been recognized in step 204 ofperforming a determination process, an output of a type of the roadsurface marking recognized, a position of the road surface markingrelative to the vehicle, and an angle of the road surface markingrelative to the vehicle are produced to the vehicle control apparatus106 a or an onboard information apparatus via a communication section109.

The following consideration should be noted. Specifically, if aplurality of objects of interest to be recognized are read in step 207for the input of a single image in step 201, processes from steps 202through 205 are repeated to complete recognition of all objects involvedof interest to be recognized. If the front camera recognition resultdata 806 contains, for example, two objects of interest to be recognizedof a pedestrian crossing and a stop line, steps from 202 to 205 arefirst performed for the recognition of the pedestrian crossing, which isthereafter followed by steps from 202 to 205 performed for therecognition of the stop line.

The operation proceeds to branch 206, if the aforementioned processesare completed for all objects of interest to be recognized read in step207. If no new image input signal is received in branch 206, theoperation is set into a wait state. If a new image input signal isreceived in branch 206, the operation returns to step 201.

FIG. 6 is a flowchart showing processes executed by the recognitionmethod evaluation section 104.

In step 301 of performing a front camera recognition result inputprocess, data of recognition result based on the image taken by thefront camera 101 is acquired from the front road surface markingrecognition section 102 a (or the road surface marking informationstorage section 111). For the recognition result data, recognitionresults of the road surface marking or the like in the image taken bythe front camera 101, luminance information of the image taken by thefront camera 101, and information on shadows on the road surface areobtained. The front camera recognition results will be described indetail with reference to processes from steps 302 to 306 that follow.

In step 302 of performing a rear camera gain value determinationprocess, the information on the shadows on the road surface in the imagetaken by the front camera 101 obtained in step 301 is analyzed and again value of the rear camera 108 is determined. This process will bedescribed in detail later.

In step 307 of performing a rear camera exposure time determinationprocess, the luminance information of the image taken by the frontcamera 101 obtained in step 301 is analyzed and an exposure time of therear camera 108 is determined. Again, this process will be described indetail later.

In step 303 of the rear camera object to be recognized determinationprocess, the object to be recognized in the rear camera 108 isdetermined based on the recognition results of the road surfacemarkings, white lines, traffic lights, and signs in the image taken bythe front camera 101 as obtained in step 301. In step 304 of performingthe rear camera process timing determination process, timing at whichthe processing for recognizing the object to be recognized in the rearcamera 108 is determined. Processes performed in steps 303 and 304 willbe described in detail later.

In step 305 of a rear camera recognition logic parameter determinationprocess, parameter values of various kinds in recognition logic of theobject to be recognized are determined based on the recognition resultsof the road surface markings and white lines within the front cameraimage and information on the shadow on the road surface within the frontcamera image obtained in step 301. The process performed in step 305will be described in detail later.

Finally in branch 306, if the recognition results of the front camera101 are not updated, the operation is set into a wait state. If therecognition results of the front camera 101 are updated, the operationreturns to step 301.

In the first embodiment described heretofore, processes of steps 302 and307, and from 303 to 305, are performed in series. Performing all thesesteps is not, however, mandatory. Rather, some processes to be adoptedare appropriately selected and combined, and thereby performed accordingto possible use conditions. In any combination, accuracy of the rearcamera 108 recognizing the road surface marking or the like can beenhanced as compared with the known art. Step 302 of the rear cameragain value determination process or step 307 of performing the rearcamera exposure time determination process is to be performed in advanceof steps from 303 to 305. This is because of the following reasons.Specifically, steps from 303 to 305 are performed on the assumption thatthe rear camera 108 has successfully imaged an object to be recognized.It is therefore necessary that imaging conditions (gain, exposure time)of the rear camera 108 be changed prior to the processes of the stepsfrom 303 to 305, so that the object to be recognized can be imaged inaccordance with the condition of brightness of surrounding areas.

In accordance with the first embodiment, the front road surface markingrecognition section 102 a is adapted to input the type of the roadsurface marking or the like recognized to the recognition methodevaluation section 104.

The road surface markings may be, as described earlier, a pedestriancrossing, a stop line, a maximum speed limit marking, a followdirection, a no U-turn marking, and the like. Each of these road surfacemarkings has unique graphic characteristics. Accordingly, differentideal image processing algorithms can apply according to different typesof the road surface marking. According to the arrangements of the firstembodiment, the type of the road surface marking or the like is firstidentified with the front camera 101 before the rear camera 108 isnotified of the type, so that the appropriate image processing algorithmcan be selected. This reduces possibility that the rear camera 108erroneously recognizes or does not recognize (fails to recognize) a roadsurface marking.

In accordance with the first embodiment, the front road surface markingrecognition section 102 a is adapted to detect the brightness of theimage taken by the front camera 101 and the shadow in the image andinput the information to the recognition method evaluation section 104.Specifically, luminance of the entire image is detected by analyzing theluminance information of the image. The recognition method evaluationsection 104 plans an adequate gain (aperture) and exposure time (shutterspeed) of the rear camera 108 and sends the data to the rear cameracontrol section 107. The rear camera control section 107 controls therear camera 108 based on the commands of the gain and exposure time ofthe rear camera 108 received from the recognition method evaluationsection 104.

Even if the image taken by the front camera 101 is, for example, toobright or too dark so that the road surface marking or the like is notclearly imaged, therefore, the rear camera 108 can take an image withthe gain and exposure appropriate for the ambient brightness. Thisallows the rear camera 108 to image the road surface marking or the likeeven more clearly, so that the road surface marking or the like can berecognized. This effect is particularly conspicuous in conditions offrequently varying ambient brightness, such as in shadows of buildingsor the like cross the road surface.

[Rear Camera Gain Value Determination Process]

Of the processes executed by the recognition method evaluation section104 shown in FIG. 2, a detailed embodiment of the step 302 of performingthe rear camera gain value determination process will be described withreference to FIGS. 4 through 10 and 22 through 26.

FIG. 7 is a detailed flowchart showing the step 302 of the rear cameragain value determination process.

In step 401 of performing a front camera shadow position referenceprocess, front camera shadow position data 408, which describes theposition of a shadow on the road surface in the front camera image, isobtained. The front camera shadow position data 408 is created by thefront road surface marking recognition section 102 a based on the imagetaken by the front camera 101. The front camera shadow position data 408is either included in the front camera recognition result data 806 orstored in the road surface marking information storage section 111 andreferred to by the recognition method evaluation section 104. A processperformed by the front road surface marking recognition section 102 afor detecting the shadow in the front camera image will be describedlater.

Referring to FIG. 10, the position of the shadow on the road surface inthe front camera image is represented by a shadow start position A (701)and a shadow end position B (702). The shadow position of FIG. 10 isthen translated to a corresponding value on a road coordinate systemshown in FIG. 22 and represented by a table format shown in FIG. 23.Referring to FIG. 22, let a point on the road surface immediately belowa center of a lens of the front camera 101 be an origin 1302. Further,let a straight line 1303, which is an optical axis of the front camera101 projected onto the road surface, be an y-axis and let a straightline 1304, which passes through the origin 1302 and extends on the roadsurface orthogonally relative to the y-axis, be an x-axis. Then, theshadow is represented by a data string as shown in FIG. 23 relative tothe road coordinate system. Specifically, the data string includes atype 1401 indicating whether the data is the shadow start position orthe shadow end position; an ID number 1402; a position (y coordinate inthe road coordinate system) 1403; a luminance mean value 1404; andshadow detection time 1406.

In step 402 of performing a rear camera shadow appearance estimationprocess, specific time is estimated, at which the shadow on the roadsurface detected by the front camera 101 appears in the field of view ofthe rear camera 108. The estimated results are written in rear camerashadow position data 407.

Step 402 will be described in detail with reference to FIGS. 24, and 25Aand 25B. FIG. 24 is a detailed flowchart showing step 402 of performingthe rear camera shadow appearance estimation process.

In step 1501 of performing a vehicle speed reference process, a vehiclespeed current value v1 is obtained.

In step 1502 of performing a shadow appearance timing calculationprocess, timing is calculated, at which the shadow on the road surfaceappears at a starting end 1602 of a road surface marking recognitionarea 1601 shown in FIG. 25A. Referring to FIG. 25A, the origin, thex-axis, and the y-axis constitute the road coordinate system describedwith reference to FIG. 22. Reference numeral 5 represents the field ofview of the rear camera 108. A marking 1601 forming part of the field ofview 5 is a road surface marking recognition area that starts at astarting end 1602. The road surface marking recognition area 1601corresponds to a range 211 shown in FIG. 5, within which the roadsurface marking appears in the screen and fades out, and over which anadequate level of resolution allowing the road surface marking to berecognized can be obtained.

Referring to FIGS. 25A and 25B, when a2 is a y-coordinate value of thestarting end 1602 of the road surface marking recognition area in theroad coordinate system; a1, a y-coordinate value of the shadow detectedby the front camera 101 in the road coordinate system; t1, time at whichthe shadow is detected; and d1, a distance between the position at whichthe front camera 101 is installed and that at which the rear camera 108is installed, a time t2, at which the shadow detected by the frontcamera 101 appears at the starting end 1602 of the road surface markingrecognition area, is given by the following equation:t2=t1+(a1+d1+|a2|)/v1

Finally in step 1503 of performing a shadow position registrationprocess, the shadow appearance timing estimated in step 1502 ofperforming the shadow appearance timing calculation process is writtenin the rear camera shadow position data 407. The rear camera shadowposition data 407 is defined as a table shown in FIG. 26. The rearcamera shadow position data 407 includes information of a type 1701indicating whether the data is the shadow start position or the shadowend position; an ID number 1702; an estimated luminance value 1703; andan estimated time of shadow appearance 1704. The estimated luminancevalue 1703 is identical to the luminance mean value 1404 of the frontcamera shadow position data of FIG. 23.

Referring back to the flowchart shown in FIG. 7, in step 403 ofperforming the rear camera gain value determination process, the rearcamera shadow position data 407, which represents the shadow appearancetiming in the rear camera field of view estimated in step 402, isobtained. Then, in accordance with the estimated luminance value of theshadow, the gain value of the rear camera 108 is determined and a rearcamera gain value schedule table shown in FIG. 8 is created.

Referring to FIG. 8, the table shows gain value change time 501indicating the specific timing at which gain values are changed and again value 502 selected at each timing of the gain value change time501. The gain value 502 is determined based on an estimated value ofluminance of the shadow that is expected to appear within the field ofview of the rear camera 108. The gain is determined with reference to atable of FIG. 27 showing luminance 1801 and a gain value 1802 of theshadow. The table shown in FIG. 27 is created by conducting an advanceexperiment to find a relationship between the luminance 1801 and thegain value 1802 of the shadow ideal for detecting the road surfacemarking.

Referring back to FIG. 7, in step 404 of writing rear camera gain valueschedule data, the rear camera gain value schedule table (FIG. 8)determined in step 403 is written in rear camera gain value scheduledata 406.

Processes shown in FIG. 9 are executed in the rear camera controlsection 107. In step 601 of performing a gain value schedule datareference process, the rear camera gain value schedule data 406 createdin the step 302 of performing the rear camera gain value determinationprocess executed by the recognition method evaluation section 104 isread at regular intervals.

A current time is next read in step 602 of performing a time referenceprocess. In branch 603, if the current time is the gain value changetime 501 described in the rear camera gain value schedule data 406, step604 of performing a rear camera gain value change process is performed.In step 604, the gain value 502 described in the rear camera gain valueschedule data 406 is transmitted to the camera control section of therear camera 108. If the current time is not the gain value change time501, the operation returns to step 601.

[Rear Camera Exposure Time Determination Process]

A detailed embodiment of step 307 of performing the rear camera exposuretime determination process, among other steps (FIG. 3) executed at therecognition method evaluation section 104 shown in FIG. 2, will bedescribed below with reference to FIGS. 28 to 30.

Processes in step 307 of performing the rear camera exposure timedetermination process among other steps (FIG. 3) executed at therecognition method evaluation section 104 will be described withreference to FIG. 28.

FIG. 28 is a detailed flowchart showing step 307 of the rear cameraexposure time determination process. In step 1803 of performing a frontcamera luminance value reference process, image luminance current valuedata 904 created by the front camera image recognition unit 102 is readto find a mean luminance value of the front camera input screen.

In step 1813 of performing a front camera luminance value reaching timecalculation process, time T2 is calculated, at which an image of theaverage luminance value of the front camera 101 obtained in step 1803appears within the field of view of the rear camera 108. To calculateit, the vehicle speed current value v1 and a current time T1 arereferred to. Then, referring to FIGS. 25A and 25B, when d1 is thedistance between the position at which the front camera 101 is installedand that at which the rear camera 108 is installed; F1, a y-coordinatevalue of an intersection point between the optical axis of the frontcamera 101 and the road surface; and R1, a y-coordinate value of anintersection point between the optical axis of the rear camera 108 andthe road surface. Then, we have:T2=T1+(F1+d1+R1)/v1

In step 1804 of performing a rear camera exposure time determinationprocess, the exposure time for the rear camera 108 is established inaccordance with the luminance value obtained in step 1803. A rear cameraexposure time schedule table shown in FIG. 29 is thereby created. Thetable defines a time 1807 indicating a timing for changing an exposuretime and an exposure time 1808 that is changed at the timing 1807. Theexposure time 1808 is determined by the average luminance value expectedwithin the field of view of the rear camera 108. The exposure time 1808is determined by referring to a table of a luminance value 1814 andexposure time 1815 shown in FIG. 31. The exposure time 1815 enabling theroad surface marking to be detected most easily according to theluminance value 1814 is determined through an advance experiment.

In step 1805 of writing a rear camera exposure time schedule data, therear camera exposure time schedule (FIG. 29) established through steps1813 and 1804 is written in rear camera exposure time schedule data1806. The rear camera control section 107 of the rear camera imagerecognition unit 103 refers to the rear camera exposure time scheduledata 1806.

Processes shown in FIG. 30 are executed at the rear camera controlsection 107.

In step 1809 of performing an exposure time schedule data referenceprocess, the rear camera exposure time schedule data 1806 createdthrough step 307 of performing the rear camera exposure timedetermination process performed at the recognition method evaluationsection 104 is read at regular intervals.

In step 1810 of performing a time reference process, the current time isread. In branch 1811, if the current time is the exposure time changetime 1807 described in the rear camera exposure time schedule data 1806,step 1812 of performing a rear camera exposure time change process isperformed. In step 1812, the exposure time 1808 described in the rearcamera exposure time schedule data 1806 is transmitted to the cameracontrol section of the rear camera 108. If the current time is not theexposure time change time 1807, the operation returns to step 1809.

[Rear Camera Object Recognition Determination Process]

A detailed embodiment of step 303 of performing the rear camera objectto be recognized determination process, among other steps (FIG. 3)executed at the recognition method evaluation section 104, will bedescribed below with reference to FIGS. 32 to 37.

FIG. 32 is a detailed flowchart showing step 303 of determining anobject to be recognized by the rear camera.

In step 1901 of performing a front camera recognition result referenceprocess, the front camera recognition result data 806, which describesthe recognition results indicating the road surface marking recognizedby the front camera image recognition unit 102, is read. The frontcamera recognition result data 806 is defined as a table shown in FIG.36. The table stores an ID number 2101, a type 2102 of the road surfacemarking, white line, traffic light, and sign recognized, a time 2103 ofrecognition, a position 2104 of the object recognized, an angle 2105 ofthe object recognized relative to the vehicle, and a degree 2106 offading in the paint of the road surface marking or white linerecognized. The position 2104 of the object recognized and the angle2105 of the object recognized relative to the vehicle are represented asshown in FIG. 33. FIG. 33 is the road coordinate system described withreference to FIG. 22. In FIG. 33, the position 2104 of the objectrecognized (in the example shown in FIG. 33, the object recognized is asign indicating a pedestrian crossing ahead, with a bicycle crossinglane) is represented by an x-coordinate value and a y-coordinate valueof a point 1907 in the object recognized closest to the vehicle. Theangle 2105 of the object recognized relative to the vehicle isrepresented by an angle formed between the y-axis and a line segment1908 which is defined as a centerline 1908 of the object recognizedextending in parallel with a white line 1910.

Of the front camera recognition result data 806, the position 2104 ofthe white line and the angle 2106 of the white line relative to thevehicle are represented as shown in FIG. 34. FIG. 34 is the roadcoordinate system described with reference to FIG. 22. Referring to FIG.34, reference numeral 1 denotes the vehicle, reference numeral 101denotes the front camera, and reference numerals 3 d, 3 e denote whitelines recognized. In FIG. 34, a point 1911 on the y-axis is defined. Thepoint 1911 is a distance of d2 ahead of the front camera. Further, whena point 1913 is an intersection point between a straight line 1912 thatpasses through the point 1911 and extends on the road surfaceorthogonally to the y-axis and the white line, the position 2104 of thewhite line is given by the x-coordinate and y-coordinate values of thepoint 1913. Further, the angle 2106 of the white line relative to thevehicle is given by an angle formed between the white line and they-axis.

How the front camera recognition result data 806 is created will bedescribed later.

In step 1902 of performing an identical road surface marking extractionprocess, data concerning the type of road surface marking and the whiteline as objects of interest to be recognized by the rear camera 108(FIG. 36) is extracted from among the front camera recognition resultdata 806 recognized by the front camera image recognition unit 102. Theextracted data serves as the objects of interest to be recognized by therear camera 108.

In step 1903 of performing a nearby road surface marking extractionprocess, the road surface markings located nearby the object recognizedare extracted as the objects of interest to be recognized by the rearcamera 108 from among the front camera recognition result data 806recognized by the front camera image recognition unit 102. For thenearby road surface markings, nearby road surface marking data 1905shown in FIG. 32 is previously registered and the nearby road surfacemarkings are extracted from the nearby road surface marking data 1905.

A table shown in FIG. 35 shows a type 2001 of road surface markings,traffic lights, and signs to be recognized by the front camera 101, atype 2002 of road surface markings located nearby the type 2001, and anassumed distance 2003 between the type 2001 and the type 2002.

In step 1904 of registering an object to be recognized, the types ofroad surface markings defined as the objects of interest to berecognized by the rear camera 108 in steps 1902 and 1903 are written inthe data 1906 on an object to be recognized by the rear camera. The data1906 on the object to be recognized by the rear camera is a table shownin FIG. 37. The table of FIG. 37 stores the following types of data: anID number 2201 of the object to be recognized; a type 2202 of the objectto be recognized; an estimated time 2203 of appearance within the fieldof view of the rear camera 108; an estimated position 2204 of the objectto be recognized appearing within the field of view of the rear camera108; an estimated angle 2205 of the object to be recognized relative tothe vehicle; and a degree 2206 of fading of paint of the road surfacemarking to be recognized.

Step 1904 of registering the object to be recognized involvesregistration of the ID number 2201, the type 2202, and the degree 2206of fading of paint among other data 1906 on an object to be recognizedby the rear camera. The rest of the data 1906 on the object to berecognized by the rear camera is registered later and thus yet to beregistered in this step 1904 of registering the object to be recognized.The ID number 2201 and the degree 2206 of fading of paint, if extractedin step 1902 of performing the identical road surface marking extractionprocess, are identical to the ID number 2101 and the degree 2106 offading of paint of the front camera recognition result data 806. Ifextracted in step 1903 of performing the nearby road surface markingextraction process, the ID number 2201 is to be newly registered and thedegree 2206 of fading is yet to be registered in the step 1904 ofregistering an object to be recognized.

[Rear Camera Process Timing Determination Process]

A detailed embodiment of step 304 of performing the rear camera processtiming determination process, among other steps (FIG. 3) executed at therecognition method evaluation section 104, will be described below withreference to FIGS. 38 to 41. FIG. 38 is a detailed flowchart showingstep 304 of performing the rear camera process timing determinationprocess.

In step 2301 of performing a vehicle speed reference process, thevehicle speed current value v1 is obtained.

In step 2302 of performing an appearance timing calculation process,timing is calculated at which the object to be recognized appears at thestarting end 1602 of the road surface marking recognition area shown inFIG. 25A. Referring to FIGS. 25A and 25B, when s1 is a y-coordinatevalue of the starting end 1602 of the road surface marking recognitionarea in the road coordinate system; a3, a y-coordinate value (2104 ofFIG. 36) of the object of interest detected by the front camera imagerecognition unit 102 in the road coordinate system; t3 (2103 of FIG.36), a time at which the object of interest is detected by the frontcamera image recognition unit 102; and d1, the distance between theposition at which the front camera 101 is installed and that at whichthe rear camera 108 is installed, a time t4, at which the objectdetected by the front camera image recognition unit 102 appears at thestarting end 1602 of the road surface marking recognition area, is givenby the following equation:t4=t3+(a3+d1+|s1|)/v1  (FIG. 41)

In step 2303 of performing an appearance position calculation process, aspecific position within the field of view of the rear camera 108 iscalculated at which the object to be recognized appears. FIG. 39 is aview defining the position of the object to be recognized by the rearcamera 108 and the angle of the same relative to the vehicle. FIG. 39represents the road coordinate system described in FIG. 22, in whichreference numeral 1 denotes a vehicle, reference numeral 108 denotes arear camera, reference numeral 5 denotes a field of view of the rearcamera 108, reference numeral 1601 denotes a road surface markingrecognition area, and reference numeral 1602 denotes a starting end ofthe road surface marking recognition area 1601. The position (x2, y2),at which an object 2306 to be recognized appears, is where a point 2306measured to determine the position of the object 2306 to be recognizedappears at the starting end 1602 (the y-coordinate value being s1) ofthe road surface marking recognition area. The angle of the object 2306to be recognized relative to the vehicle is an angle r2 formed between aline segment 2307, which is defined as a centerline 2307 of the object2306 to be recognized extending in parallel with a white line, and they-axis. Assume that the position (2104 of FIG. 36) of the object to berecognized of the front camera recognition result data 806 is (x1, y1),the angle 2105 relative to the vehicle is r1, and the distance betweenthe position at which the front camera 101 is installed and that atwhich the rear camera 108 is installed is d1. It is further assumed thatthe road is straight and the vehicle has a steering angle of 0. Then, wehave:x2=x1±(y1+d1+|s1|)*tan(r1) (positive or negative is selected for thesign ± according to whether r1 is positive or negative)y2=s1r2=r1

If the object to be recognized is a white line, the position and theangle relative to the vehicle are defined as shown in FIG. 40 and theposition of appearance within the field of view of the rear camera iscalculated. FIG. 40 is the road coordinate system described in FIG. 22,in which reference numeral 1 denotes a vehicle, reference numeral 108denotes a rear camera, reference numeral 5 denotes a field of view ofthe rear camera 108, reference numeral 1601 denotes the road surfacemarking recognition area described with reference to FIG. 16, andreference numeral 1602 denotes a starting end of the road surfacemarking recognition area 1601. Further, a point 2308 is defined on they-axis and at a distance of d3 rearward of the rear camera 108. Herein,let reference numeral 2307 be an intersection point between a straightline 2309 on the road surface passing through the point 2308 andextending orthogonally to the y-axis and a white line 2307. Then, theposition of the white line (x4, y4) is the x-coordinate and y-coordinatevalues of the intersection point 2310. An angle r4 of the white linerelative to the vehicle is expressed by an angle formed between thewhite line and the y-axis. Assume that the position (2104 of FIG. 36) ofthe object to be recognized (white line) of the front camera recognitionresult data 806 is (x3, y3), the angle 2105 relative to the vehicle isr3, and the distance between the position at which the front camera 101is installed and that at which the rear camera 108 is installed is d1.It is further assumed that the road is straight and the vehicle has asteering angle of 0. Then, we have:x4=x3±(y3+d1+d3)*tan(r3) (positive or negative is selected for the sign± according to whether r3 is positive or negative)y4=−d1−d3r4=r3

Finally in step 2304 of registering an object to be recognized, thetiming of the object to be recognized appearing within the field of viewof the rear camera calculated in step 2302 and the position of theobject to be recognized appearing within the field of view of the rearcamera calculated in step 2303 are written in the time 2203 ofappearance, the position 2204, and the angle 2205 of the table (FIG. 37)of the data 1906 on the object to be recognized by the rear camera. Ifthe object to be recognized is a white line, the time 2203 of appearanceis the current time.

If the object to be recognized is extracted in step 1903 of performing anearby road surface marking extraction process, the time 2203 ofappearance is as follows. Specifically, the timing is calculated atwhich the front camera object to be recognized (2001 of FIG. 35) appearswithin the field of view of the rear camera. The time 2203 of appearanceshould allow for the assumed distance (2003 of FIG. 35) between thefront camera object to be recognized (2001 of FIG. 35) and the nearbyroad surface marking type (2002 of FIG. 35). The position 2204 and theangle 2206 are to be yet to be registered.

[Rear Camera Recognition Logic Parameter Establishment Process]

A detailed embodiment of step 305 of performing the rear camerarecognition logic parameter determination process, among other steps(FIG. 3) executed at the recognition method evaluation section 104, willbe described below with reference to FIGS. 42 to 47. FIG. 42 is adetailed flowchart showing step 305 of performing the rear camerarecognition logic parameter determination process.

In step 2401 of referencing data on an object to be recognized by therear camera, contents (table of FIG. 37) registered in the data 1906 onthe object to be recognized by the rear camera are read.

In subsequent step 2402 of performing a rear camera recognition areaadjustment process, an x-coordinate value of the position (2204 of FIG.37) of the data 1906 on the object to be recognized by the rear camerais converted to a corresponding value in a screen coordinate system. Thescreen coordinate system refers to a coordinate system having an originO at an upper left corner 2408 of a rear camera input screen 2407, au-axis extending in a width direction 2409 of the screen 2407, and av-axis extending in a height direction 2410 of the screen 2407. In step2402, a rear road surface marking appearance position 2412 (u6 of the ucoordinate value) of an object 2411 to be recognized on the rear camerainput screen 2407 is first calculated. An ordinary recognition area isindicated by a dotted line 2413. For the object 2411 that is to appearat the appearance position 2412, an area equivalent to width of theobject to be recognized is corrected about the rear road surface markingappearance position 2412. Specifically, a recognition area 2414 of theobject to be recognized after correction inside a solid line iscorrected as the recognition area of the object to be recognized. Aconversion table shown in FIG. 44 is prepared in advance for conversionfrom the road coordinate system to the screen coordinate system. Thecoordinate system conversion is made by referring to this table of FIG.44. The table of FIG. 44 stores an x-coordinate value 2415 and ay-coordinate value 2416 in the road coordinate system, a u-coordinatevalue 2417 and a v-coordinate value 2418 in the screen coordinatesystem, the u-coordinate value 2417 and the v-coordinate value 2418corresponding to the x-coordinate value 2415 and the y-coordinate value2416.

In step 2403 of performing a characteristic quantity threshold valuedetermination process, a threshold value for extraction of acharacteristic quantity of the road surface marking is established byusing the degree (2206 of FIG. 37) of fading of the data 1906 on theobject to be recognized by the rear camera and the rear camera shadowposition data 407 at the timing at which the object to be recognizedappears within the field of view of the rear camera 108 (table of FIG.26).

Methods for determining the degree of fading and the characteristicquantity extraction threshold value will be described with reference toFIG. 45. Referring to FIG. 45, reference numeral 2419 shows how the roadsurface marking is seen when the degree of fading is low. Referencenumeral 2420 shows how the road surface marking is seen when the degreeof fading is high. Reference numerals 2421, 2422 represent changes inthe luminance value of the portion of the road surface marking. With alow degree of fading, an outline of the road surface marking portion isextracted on the assumption that the change in the luminance value of aportion 2423 is precipitous. For a high degree of fading, on the otherhand, the outline of the road surface marking portion is extracted onthe assumption that the change in the luminance value of a portion 2424is moderate.

A method for determining the characteristic quantity extractionthreshold value in accordance with presence of a shadow on the roadsurface at the timing, at which the object to be recognized appearswithin the field of view of the rear camera 108 will be described withreference to FIG. 46. The time at which the shadow starts and that atwhich the shadow ends (1704 of FIG. 26) of the rear camera shadowposition data 407 are first referred to. The road surface marking isseen as 2425 if there is no shadow on the rear camera input screen atthe time (2203 of FIG. 37), at which an object to be recognized 2425appears in the recognition area. The road surface marking is seen as2427 if there is a shadow 2432 on the rear camera input screen.Reference numerals 2428, 2429 represent changes in the luminance valueof the portion of the road surface marking. With no shadows, the outlineof the road surface marking portion is extracted on the assumption thatthe maximum of the luminance value of a portion 2430 is high. If thereis a shadow, on the other hand, the outline of the road surface markingportion is extracted on the assumption that the maximum of the luminancevalue of a portion 2431 is low.

Finally in step 2405 of performing a recognition parameter registrationprocess, the parameter values established through steps 2402 and 2403are registered in the recognition parameter data 2406. The recognitionparameter data 2406 is a table shown in FIG. 47. The table of FIG. 47records: an ID number 2432 of the object to be recognized; au-coordinate value 2433 on the left end of a rectangle of therecognition area (2414 of FIG. 43); a u-coordinate value 2434 on theright end of the rectangle of the recognition area (2414 of FIG. 43); anestimated value 2435 (2423, 2424 of FIG. 45) of changes in the luminancevalue between the road surface and the road surface marking; and anestimated value 2436 (2430, 2431 of FIG. 46) of the maximum of theluminance value of the road surface marking portion. The ID number 2432corresponds to the ID number (2201 of FIG. 37) of the data 1906 on theobject to be recognized by the rear camera.

Processes performed by the front road surface marking recognitionsection 102 a will be described below with reference to FIG. 11. FIG. 11is a flowchart showing the processes performed by the front road surfacemarking recognition section 102 a.

In step 801 of performing an image input process, the image taken by thefront camera 101 is obtained.

In step 802 of performing an image luminance statistical process,statistical data of the luminance value of the input image isaccumulated and analyzed, and written in image luminance statisticaldata 804. Step 802 will be described in detail later.

In step 803 of performing a shadow position recognition process, it isdetermined whether or not there is a shadow on the road surface of theinput screen. Results of the determination are written in the frontcamera shadow position data 408. Step 803 will be described in detaillater.

In step 805 of selecting an object to be recognized, the type of theobject to be recognized is selected. Step 807 to be described below isperformed to recognize each of the objects of interest to be recognized.The object to be recognized is selected by the vehicle control apparatus106 a, the onboard information apparatus 106 b, or the front cameraimage recognition unit 102.

In step 807 of performing an object recognition process, a process isperformed for detecting the object to be recognized selected in step805. Details of this process will be described later.

Finally in branch 807, if no new image input signal is received, theoperation is set into a wait state. If a new image input signal isreceived in branch 807, the operation returns to step 801.

[Image Luminance Statistical Process]

Step 802 of performing the image luminance statistical process, amongother steps (FIG. 11) performed by the front road surface markingrecognition section 102 a, will be described in detail below withreference to FIGS. 12 to 18. FIG. 12 is a flowchart showing processesperformed in step 802 of performing the image luminance statisticalprocess.

In step 901 of performing an image luminance acquisition process, theluminance value of the input image is obtained and written in imageluminance current value data 904 and image luminance accumulated data903. Referring to FIG. 14, when the luminance value of the input imageis to be acquired, a distribution of the luminance values of an area1001 including the road surface is acquired as a distribution ofluminance and frequency as shown in FIG. 13. The data is accumulated inthe image luminance accumulated data 903.

In step 902 of performing a luminance distribution update process, theimage luminance accumulated data 903 acquired and updated in step 901 isloaded, and an image luminance statistical table as shown in FIG. 15 iscreated and written in the image luminance statistical data 804.Referring to FIG. 15, reference numeral 1002 denotes the luminance ofthe image and reference numeral 1003 indicates the condition of the roadsurface under the corresponding luminance. The states of the roadsurface includes a state 1004 where there is a shadow on the roadsurface, a state 1005 where there is a shadow on a white road surfacemarking, a state 1006 where there is no shadow on the road surface, anda state 1007 where there is no shadow on the white road surface marking.

The condition of the road surface is evaluated as follows. Specifically,if the distribution of luminance frequency has four peaks (1101 to 1104)as shown in FIG. 16 in the image luminance accumulated data 903 loadedin step 902, it is determined that there is a portion having a shadowand a portion not having a shadow on the road surface (typically in thedaytime with sunshine). Spots (1105 to 1107) having the lowest frequencyof luminance are extracted between each pair of adjacent peaks. Theimage luminance statistical table shown in FIG. 15 is then created withluminance values (a, b, c) at corresponding spots used as boundaries.

Further, referring to FIG. 17, if four peaks are not formed in thedistribution of luminance frequency in the image luminance accumulateddata 903, it is determined that no sunshine is available even in thedaytime, or it is nighttime. The image luminance statistical table asshown in FIG. 15 is then unknown.

[Shadow Position Recognition Process]

Step 803 of performing the shadow position recognition process, amongother steps (FIG. 11) performed by the front road surface markingrecognition section 102 a, will be described in detail below withreference to FIGS. 19 to 21. FIG. 19 is a flowchart showing processesperformed in step 803 of performing the shadow position recognitionprocess.

In step 1201 of performing an image luminance acquisition process, thefront camera input screen is divided into a plurality of small areas1108 and a mean luminance in each area 1108 is calculated.

In subsequent step 1202 of performing a shadow position determinationprocess, the image luminance statistical data 804 created in step 802 ofperforming the image luminance statistical process is loaded. Acomparison is then made between the mean luminance of each area 1108 ofFIG. 18 acquired in step 1201 and the luminance value (1002 of FIG. 15)of the image luminance statistical table of the image luminancestatistical data 804. A specific area or areas of the front camera inputscreen of FIG. 18 are thereby determined to be a shadow and the shadowstart position 701 and the shadow end position 702 of FIG. 10 areextracted.

In step 1203 of performing conversion to road coordinate system, theshadow start position 701 and the shadow end position 702 extracted instep 1202 are translated to corresponding values in the road coordinatesystem. The shadow start position 701 and the shadow end position 702extracted in step 1202 are in the screen coordinate system.Specifically, referring to FIG. 20, the screen coordinate system has anorigin O at an upper left corner 1205 of the front camera input screen,a u-axis extending in a width direction 1206 of the screen, and a v-axisextending in a height direction 1207 of the screen. The road coordinatesystem is, on the other hand, a coordinate system shown in FIG. 22 asdescribed earlier. Conversion from the screen coordinate system to theroad coordinate system is made by referring to a conversion table shownin FIG. 21 prepared in advance. Referring to FIG. 21, reference numeral1208 denotes a u-coordinate value and reference numeral 1209 denotes av-coordinate value, respectively, in the screen coordinate system. Thesevalues are keyed to an x-coordinate value 1210 and a y-coordinate value1211, respectively, in the road coordinate system.

Finally in step 1204 of performing a shadow position registrationprocess, the shadow start position 701 and the shadow end position 702,which have been translated to the corresponding values in the roadcoordinate system in step 1203, are written in the front camera shadowposition data 408. The front camera shadow position data 408 is in aform of a table shown in FIG. 23 as described earlier. The coordinatevalues translated in step 1203 are registered in the position 1403 andthe luminance values of portions of no road surface markings areregistered in the luminance mean value 1404.

[Object of Interest Recognition Process]

Step 807 of performing the object recognition process, among other steps(FIG. 11) performed by the front road surface marking recognitionsection 102 a, will be described in detail below. If the object to berecognized selected in step 805 is a road surface marking or a whiteline, the same steps as those from 202 to 206 of a road surface markingrecognition function 106 performed by the rear camera 108 shown in FIG.3 are performed. During this time, in step 203 of performing the roadsurface marking characteristic quantity extraction process as shown inFIG. 3, the recognition parameter data 2406 is not loaded. If the objectto be recognized selected in step 805 is a traffic light, a rectangularoutline of the traffic light is detected through pattern matching in theinput screen. Three circular shapes are then detected within thedetected rectangle through pattern matching. Next, color information inthe circular shape detected is obtained. If the color informationobtained corresponds to any of red, yellow, and blue, it is thendetermined that the object is the traffic light.

If the object to be recognized selected in step 805 is a sign, patternmatching is performed to detect the shape of the sign to be recognized.Pattern matching is then performed for characters marked on the signdetected. If there is a match in the characters on the sign, it is thendetermined that the sign detected is one of the objects of interest tobe recognized.

Other embodiments will be described below with reference to FIGS. 48through 52.

Second Embodiment

FIG. 48 shows a hardware block diagram of a system for recognizing anenvironment surrounding a vehicle according to a second embodiment.Major differences from the first embodiment described with reference toFIGS. 1 through 47 include the following. Specifically, a front cameraimage recognition unit 102 is disposed inside a front camera 101; and arear camera image recognition unit 103 is disposed in another vehiclecontrol function 2510 a or onboard information function 2510 b.Accordingly, the same processes for recognizing and evaluating thesurrounding environment as those of the first embodiment apply unlessotherwise noted. Differences from the first embodiment will be describedbelow.

The front camera 101 includes a lens 2501, an imaging device (CCD) 2502,a CPU 2503, and a memory (not shown). The front camera 101 achieves thefunction of the front camera image recognition unit 102 using the CPU2503 and the memory. A rear camera 108, on the other hand, includes alens 2504 and an imaging device (CCD) 2505.

The front camera 101 is connected to a running control function 2510 aor an onboard information function 2510 b (hereinafter referred to as“vehicle control apparatus 2506”) via a CAN 2507 to permit dataexchanged therebetween. The vehicle control apparatus 2506 has afunction of the rear camera image recognition unit 103, in addition tothose of the running control function 2510 a and the onboard informationfunction 2510 b.

The rear camera 108 and the vehicle control apparatus 2506 are connectedvia an image signal line 2509 and a dedicated signal line 2508. Theimage taken by the rear camera 108 is transmitted to the rear cameraimage recognition unit 103 of the vehicle control apparatus 2506 overthe image signal line 2509. A signal for controlling the rear camera 108is transmitted from the rear camera image recognition unit 103 of thevehicle control apparatus 2506 over the dedicated signal line 2508.

The arrangement according to the second embodiment allows, if applied toa case involving a large volume of data being transmitted between therear camera image recognition unit 103 and the running control function2510 a or the onboard information function 2510 b, a large volume ofdata to be transmitted using an internal bus of the vehicle controlapparatus 2506. This offers a good system performance.

Third Embodiment

FIG. 49 shows a hardware block diagram of a system for recognizing anenvironment surrounding a vehicle according to a third embodiment. Majordifferences from the first embodiment described with reference to FIGS.1 through 47 include the following. Specifically, a front camera imagerecognition unit 102 is disposed inside a front camera 101; and a rearcamera image recognition unit 103 is disposed inside a rear camera 108.Accordingly, the same processes for recognizing and evaluating thesurrounding environment as those of the first embodiment apply unlessotherwise noted. Differences from the first embodiment will be describedbelow.

The front camera 101 shares the same arrangement with that of the secondembodiment, except that the front camera 101 according to the thirdembodiment is connected to the rear camera 108 via a dedicated signalline 2609.

The rear camera 108 includes a lens 2504, an imaging device (CCD) 2505,a CPU 2608, and a memory (not shown). The rear camera 108 achieves thefunction of the rear camera image recognition unit 103 using the CPU2608 and the memory.

The front camera 101 has a CPU 2503 connected to the CPU 2608 of therear camera 108 with the dedicated signal line 2609. The CPU 2503 andthe CPU 2608 exchange data therebetween. Further, a vehicle controlapparatus 2606 mounted with a running control function 2510 a and anonboard information function 2510 b, the front camera 101, and the rearcamera 108 transmit data to each other via a CAN 2607.

The arrangement according to the third embodiment offers a good systemperformance when applied to a case involving a large processing load onthe front camera image recognition unit 102 and the rear camera imagerecognition unit 103.

Fourth Embodiment

FIG. 50 shows a hardware block diagram of a system for recognizing anenvironment surrounding a vehicle according to a fourth embodiment.Major differences from the first embodiment described with reference toFIGS. 1 through 47 include the following. Specifically, a front cameraimage recognition unit 102 and a rear camera image recognition unit 103are disposed inside a front camera 101. Accordingly, the same processesfor recognizing and evaluating the surrounding environment as those ofthe first embodiment apply unless otherwise noted. Differences from thefirst embodiment will be described below.

The front camera 101 includes a lens 2501, an imaging device (CCD) 2502,a CPU 2703, and a memory (not shown). The front camera 101 achieves thefunctions of the front camera image recognition unit 102 and the rearcamera image recognition unit 103 using the CPU 2703 and the memory. Therear camera 108 is arranged in the same manner as in the secondembodiment (FIG. 25), except that the rear camera 108 is connected tothe front camera 101 with an image signal line 2709 and a dedicatedsignal line 2708.

The front camera 101 and the rear camera 108 are connected with theimage signal line 2709 and the dedicated signal line 2708. The imagetaken by the rear camera 108 is transmitted to the rear camera imagerecognition unit 103 in the front camera 101 over the image signal line2709. A signal for controlling the rear camera 108 is transmitted fromthe rear camera image recognition unit 103 in the front camera 101 tothe rear camera 108 over the dedicated signal line 2708.

A vehicle control apparatus 2610 mounted with a running control function2510 a and an onboard information function 2510 b and the front camera101 are connected with a CAN 2507, by which data can be mutuallyexchanged therebetween.

The arrangement according to the fourth embodiment offers good systemperformance when applied to a case involving a large volume of datatransferred between the front camera image recognition unit 102 and therear camera image recognition unit 103.

Fifth Embodiment

FIG. 51 shows a hardware block diagram of a system for recognizing anenvironment surrounding a vehicle according to a fifth embodiment. Majordifferences from the first embodiment described with reference to FIGS.1 through 47 include the following. Specifically, a front camera imagerecognition unit 102 and a rear camera image recognition unit 103 aredisposed inside a rear camera 108. Accordingly, the same processes forrecognizing and evaluating the surrounding environment as those of thefirst embodiment apply unless otherwise noted. Differences from thefirst embodiment will be described below.

The front camera 101 includes a lens 2501 and an imaging device (CCD)2502. The rear camera 108 includes a lens 2504, an imaging device 2505,a CPU 2803, and a memory (not shown). The rear camera 108 achieves thefunctions of the front camera image recognition unit 102 and the rearcamera image recognition unit 103 using the CPU 2803 and the memory.

The front camera 101 and the rear camera 108 are connected with an imagesignal line 2809. The image taken by the front camera 101 is transmittedto the front camera image recognition unit 102 in the rear camera 108over the image signal line 2809. A vehicle control apparatus 2610mounted with a running control function 2510 a and an onboardinformation function 2510 b and the rear camera 108 are connected with aCAN 2507, by which data can be mutually exchanged therebetween.

The arrangement according to the fifth embodiment offers good systemperformance when applied to a case involving a large volume of datatransferred between the front camera image recognition unit 102 and therear camera image recognition unit 103.

Sixth Embodiment

FIG. 52 shows a hardware block diagram of a system for recognizing anenvironment surrounding a vehicle according to a sixth embodiment. Majordifferences from the first embodiment described with reference to FIGS.1 through 47 include the following. Specifically, a front camera imagerecognition unit 102 and a rear camera image recognition unit 103 aredisposed inside a vehicle control apparatus 2906. Accordingly, the sameprocesses for recognizing and evaluating the surrounding environment asthose of the first embodiment apply unless otherwise noted. Differencesfrom the first embodiment will be described below.

The front camera 101 includes a lens 2501 and an imaging device 2502.The rear camera 108 includes a lens 2504 and an imaging device (CCD)2505. The vehicle control apparatus 2906 has the functions of the frontcamera image recognition unit 102 and the rear camera image recognitionunit 103, in addition to those original functions of a running controlfunction 2510 a or an onboard information function 2510 b.

The front camera 101 and the vehicle control apparatus 2906 areconnected together with an image signal line 2911. The image taken bythe front camera 101 is transmitted to the front camera imagerecognition unit 102 in the vehicle control apparatus 2906 over theimage signal line 2911. The rear camera 108 and the vehicle controlapparatus 2906, on the other hand, are connected together with an imagesignal line 2909 and a dedicated signal line 2908. The image taken bythe rear camera 108 is transmitted to the rear camera image recognitionunit 103 in the vehicle control apparatus 2906 over the image signalline 2909. A signal for controlling the rear camera 108 is transmittedto from the rear camera image recognition unit 103 in the vehiclecontrol apparatus 2906 to the rear camera 108 over the dedicated signalline 2908.

The arrangement according to the sixth embodiment offers good systemperformance when applied to a case involving a large volume of datatransferred across the front camera image recognition unit 102, the rearcamera image recognition unit 103, and the running control function 2510a or the onboard information function 2510 b.

A method for inspecting to determine if the present invention isoperational will be described below.

The vehicle 1 with the arrangement as shown in FIG. 1 runs on a road. Acheck is made on the recognition rate of the object to be recognized onthe road, which is obtained by the rear camera image recognition unit103 during normal operation by measuring the operation in the vehiclecontrol apparatus 106. Then, with the lens of the front camera 101covered in the arrangement shown in FIG. 1, the vehicle 1 runs on thesame road, at the same speed, and in the same running manner as in theabove. The recognition rate of the object to be recognized on the road,which is obtained by the rear camera image recognition unit 103, ismeasured. The recognition rate under normal operation is then comparedagainst that with the lens of the front camera 101 covered. If therecognition rate under normal operation is higher than the recognitionrate with the lens of the front camera 101 covered, it may be determinedthat the present invention is operational in the arrangement shown inFIG. 1.

Another possible method for inspecting to determine if the presentinvention is operational is as follows. Specifically, the vehicle 1 withthe arrangement as shown in FIG. 1 runs on a road having significantchanges in luminance and an image taken by the rear camera 108 isacquired. Then, the vehicle 1 is run on a road having the samesignificant changes in luminance as above with the lens of the frontcamera 101 covered and an image taken by the rear camera 108 isacquired. The image acquired when the lens of the front camera 101 isnot covered is compared against that acquired when the lens of the firstcamera 101 is covered. If timing of gain adjustment and exposure controladjustment is earlier in the image of the former case, then it may bedetermined that the rear camera control section 107 of the presentinvention is operational in the arrangement shown in FIG. 1.

Seventh Embodiment

Each of the first to sixth embodiments is concerned with the arrangementusing the front camera and the rear camera. Each embodiment may includea plurality of cameras, each having a unique field of view and imagingthe same object of interest at unique timing. Embodiments will bedescribed below with reference to FIGS. 53 and 54, in which a pluralityof cameras is disposed to face the same direction in the vehicle.

Referring to FIG. 53, a first front camera 3001 is disposed with anangle of depression that allows an image at a far site forward of thevehicle to be taken. A second front camera 3002 is disposed such thatthe second front camera 3002 can image a site closer to the vehicle thanthe image taken by the first front camera 3001, preferably at a siteimmediately near the vehicle. A second front camera image recognitionunit 3004 detects the type, position, angle, and the like of a roadsurface marking and a white line in the image taken by the second frontcamera 3002. Results of the detection are transmitted to a vehiclecontrol apparatus 106 a or an onboard information apparatus 106 b.

A first front camera image recognition unit 3003 detects the type,position, angle, and the like of a road surface marking, a white line, atraffic signal, and a sign in the image taken by the first front camera3001. A recognition method evaluation section 3005 receives an outputfrom the first front camera image recognition unit 3003 representingrecognition results concerning the road surface marking, the white line,the traffic signal, and the sign located forwardly of the vehicle. Therecognition method evaluation section 3005 then establishes arecognition method in the second front camera image recognition unit3004 and transmits the recognition method to the second front cameraimage recognition unit 3004.

The first front camera image recognition unit 3003 analyzes luminanceinformation of the image taken by the first front camera 3001 anddetects luminance of the entire image or the position of a shadow on theroad surface. The first front camera image recognition unit 3003 thentransmits the results to the recognition method evaluation section 3005.The recognition method evaluation section 3005 schedules an adequategain and exposure time for the second front camera 3002 and transmitsthe schedule to a second front camera control section 3006. Inaccordance with the schedule of the gain and exposure time for thesecond front camera 3002 received from the recognition method evaluationsection 3005, the second front camera control section 3006 controls thesecond front camera 3002. The first front camera 3001, which images aview far forward of the vehicle, is advantageous in identifying trend inthe entire image. The second front camera 3002, which images a viewimmediately near the vehicle on the other hand, is advantageous indetecting with high accuracy the position and angle of the road surfacemarking and white line to be recognized.

Processes performed by the first front camera image recognition unit3003 are identical to those performed by the front road surface markingrecognition section 102 a shown in FIG. 2. Processes performed by thesecond front camera image recognition unit 3004 are identical to thoseperformed by the front road surface marking recognition section 102 ashown in FIG. 2. Processes performed by the second front camera controlsection 3006 are identical to those performed by the rear camera controlsection 107 shown in FIG. 2. Unlike the case with the rear camera,however, no inversion occurs in a left-and-right positional relationshipbetween images taken by the two cameras and coordinate conversion isdifferent from that of the embodiment shown in FIG. 1.

FIG. 54 shows a hardware block diagram that achieves the embodimentshown in FIG. 53. The first front camera 3001 includes a lens 3102 andan imaging device (CCD) 3103. The second front camera 3002 includes alens 3105 and an imaging device 3106. The second front camera 3002 isdisposed in a headlight 3108.

A vehicle control apparatus or an onboard information apparatus(hereinafter referred to as “onboard control apparatus or the like”)3107 has mounted therein the first front camera image recognition unit3003, the second front camera image recognition unit 3004, therecognition method evaluation section 3005, the second front cameracontrol section 3006, and a running control function 2510 a or anonboard information function 2510 b. The first front camera 3001 and theonboard control apparatus or the like 3107 are connected together withan image signal line. The image taken by the first front camera 3001 istransmitted to the first front camera image recognition unit 3003 in theonboard control apparatus or the like 3107 over an image signal line 2.The second front camera 3002 and the onboard control apparatus or thelike 3107 are connected together with an image signal line 2909 and adedicated signal line 2908. The image taken by the second front camera3002 is transmitted to the second front camera image recognition unit3004 inside the onboard control apparatus or the like 3107 over thededicated signal line 2908. A signal controlling the second front camera3002 is transmitted from the second front camera control section 3006inside the onboard control apparatus or the like 3107 to the secondfront camera 3002 over the dedicated signal line 2908.

The road surface marking recognition system described in thespecification is applicable, in a vehicle mounted with a plurality ofcameras, to a preventive safety system that prevents collision withother vehicles and provides driving support by recognizing vehiclesrunning near the host vehicle other than road surface markings.

1. An apparatus for recognizing an environment surrounding a vehicle,the apparatus comprising: a first image input unit for imaging anenvironment surrounding the vehicle; an object recognition unit forrecognizing an object by processing an image inputted thereto from thefirst image input unit, the object being present in the environmentsurrounding the vehicle; and a second image input unit disposed on thevehicle, wherein the second image input unit uses a result of therecognition of the object present in the environment surrounding thevehicle to adjust a parameter for the first image input unit.
 2. Theapparatus for recognizing an environment surrounding a vehicle accordingto claim 1, wherein the second image input unit uses the result of therecognition of the object present in the environment surrounding thevehicle to determine a type of the object recognized by the objectrecognition unit and a start timing of the object recognition unit. 3.The apparatus for recognizing an environment surrounding a vehicleaccording to claim 1, wherein information on luminance of an image takenby the second image input unit is used to adjust a gain of the firstimage input unit.
 4. The apparatus for recognizing an environmentsurrounding a vehicle according to claim 1, wherein the information onthe luminance of the image taken by the second image input unit is usedto control an exposure of the first image input unit.
 5. The apparatusfor recognizing an environment surrounding a vehicle according to claim1, wherein information on a position and an angle of the objectrecognized by the first image input unit disposed forwardly of thevehicle is used to determine an area to be processed on a screen in animage recognition unit for recognizing an image of the second imageinput unit disposed on rearward of the vehicle.
 6. The apparatus forrecognizing an environment surrounding a vehicle according to claim 3,wherein information on luminance and a position of a shadow on a roadsurface recognized by the first image input unit disposed forwardly ofthe vehicle is used to adjust a gain of the second image input unitdisposed on rearward of the vehicle.
 7. An apparatus for recognizing anenvironment surrounding a vehicle, the apparatus comprising: a pluralityof image input units for imaging an environment surrounding the vehicle;and an object recognition unit for recognizing an object present in theenvironment surrounding the vehicle by processing an image inputtedthereto from the image input units wherein the object recognition unitincludes a processing section having a single memory for processingimages taken by the plurality of image input units.